On Fri, Aug 9, 2024 at 5:44 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +You have to pay attention to the progress of the transcription if you want all of your spoken words captured. + +If the transcription halts, it is best to deactivate the plugin, activate it, and resume the dictation. +Great care must be taken to pronounce the first word of the sentence loudly so that it will be recorded; otherwise, this first word will likely not be recorded. +This problem of omitted words is most acute when there has been a pause in the dictation. + +The software does not automatically insert punctuation marks. +You have to vocalize the name of the punctuation mark where it is required. +You also have to utilize the built-in new-line command to start new lines. +We have combined the period command with the new line command to create a new command with the voice trigger of `new sentence`. + +You have to develop the habit of using this command if you like to write one sentence per line. +This latter form of writing is very useful for first drafts because it greatly eases the shuffling of sentences in a text editor during rewriting. +This form of writing is also very compatible with version control systems like git because the changes can be tracked by line number. + +The practical limit of on the number of commands it set by the trouble you are willing to tolerate in scrolling up and down the list of commands. you mean "is set by" rather than "it set by" — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC7TBXUJBJCLOT7EYMDZQVA5TAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGE3TCOBWHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:10 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +Our voice-triggered snippets still require customizing to the problem at hand via use of the keyboard, but their insertion by voice command saves time. + +To facilitate the use of voice commands in Jupyter notebooks, we have developed libraries of voice-triggered snippets for use in Markdown or code cells with the *Voice-In Plus* plugin. +We are building on our experience with tab-triggered code snippets in text editors ***@***.*** and domain-specific code snippet libraries for Jupyter ***@***.***. +We have made libraries of these voice-triggered snippets for several of the popular modules of the scientific computing stack for Python. +These voice-triggered snippets are another tool for software engineering that complement existing tools for enhancing productivity. + +## Methods and Materials + +### Hardware +We used a 2018 15-inch MacBook Pro laptop computer. +It had 32 gigabytes of RAM and one Radeon Pro 560X 4 GB GPU. +We used the laptop's built-in microphone to record dictation while sitting or standing up to 20 feet (ca. 6 m) away from the computer. + +### Installation of Voice In Plus +We used the *Voice In* plugin provided by Dictanote Inc. It's good that you have the hyperlinks there, but a citation in the references would be good as well. — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC2GL7I2P5AT6N434U3ZQU463AVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGEYTCMBTHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:13 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +We selected **English (United States)** from the 15 dialects of English. +The English variants include dialects from Western Europe, Africa, South Asia, and Southeast Asia; many languages other than English are also supported. + +Next, we set a keyboard shortcut for activating the plugin. +We selected command-L on our Mac because this shortcut was not already in use. +A list of Mac shortcuts can be found [here](https://support.apple.com/en-us/102650). +The second customization was the limit of the customizations that we could make in the free version of *Voice In*. + +The use of customized text replacements is available in *Voice In Plus*. +*Voice In Plus* is activated by purchasing a $39 annual subscription through the **Subscription** submenu in the *Voice In Settings* sidebar of the *Voice In Options* web page. +Monthly and lifetime subscription options are available. +Only one subscription was required to use *Voice In Plus* in both web browsers. +The library is synched between the browsers. + +After a subscription was purchased, we obtained access to the **Custom Commands** in the *Voice In Settings* sidebar. +We used the **Add Command** button to enter a voice-trigger to start a new sentence when writing one sentence per line {ref}`fig:newSentence`. Some discussion about why it might be counterproductive to use longer command phrases than "new sentence" or "new paragraph" would be good. For example, why did you not say, "Voice Plugin Plus new sentence"? — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC72ODUNTPQOTRA7DLLZQU5G5AVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGEYTINBZG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:21 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +The transcription of the spoken words appear momentarily in a transitory transcript that hovers about the text box. +We mapped the misinterpreted words to the desired text replacement. +For example, we added the mapping of `open syfy` to `<open:https://www.scipy2024.scipy.org/schedule>` to open the webpage for the schedule of SciPy 2024 when we said, "open SciPy 2024". + +The **Export** button opened a text box with the custom commands in CSV file format. +All the contents of the text box were selected, copied, and pasted into a local CSV file using either the text editor TextMate or Emacs version 29.3. +The **Synch** button was used to synchronize devices. + +A GUI shows all the voice triggers and their text replacements immediately below the row of buttons just mentioned above. +Each row in the GUI has an edit icon and a delete icon. +The edit icon opens a pop-up menu similar to the pop-up menu invoked by the **Add Command** button. + +### Construction of the snippet libraries +Some of our voice snippets had already been used for a year to compose prose using dictation. +These snippets are in modular CSV files to ease their selective use. +The contents of these files can be copied and pasted into the `bulk add` text area of the Voice In Plus configuration GUI. Some examples would be good to see. — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC7JOEFGXFHCNXSI3W3ZQU6FHAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGEZDOOJVHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:21 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +We stored each question, answer, and explanation triple in a tuple because tuples are immutable. +We stored the tuples in a list because lists are sortable. +The quiz randomizes the order of the questions upon restart, ensuring a fresh and challenging experience every time. +When you fail to answer a question correctly, the quiz provides feedback, giving you the opportunity to learn from your mistakes and improve. +Recycling the wrongly answered questions builds the recall of the correct answers. +The number of questions in a quiz was limited to 40 to avoid exhausting the user. + +### Availability of the libraries and quizzes +We tested the libraries using Jupyter Lab version 4.2 and Python 3.12 installed from MacPorts. +All libraries are available at MooersLab on GitHub for download. + +## Results + +First, we describe the contents of the snippet libraries and the kinds of problems they solve. +We group the libraries into several categories to simplify their explanation. +Second, we describe the deployment of the snippet libraries for Voice In Plus (VIP), an automated speech recognition plugin for the web browsers Google Chrome and Microsoft Edge. You are repeating yourself here, you already introduced Voice In Plus — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZECZBXAAYA4MFPO6HBWLZQU6HFAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGEZDQOBXHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:24 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > + +The libraries are made available in a modular format so the user can select the most valuable commands for their workflow. +In general, our libraries are broad in scope, so they meet the needs of most users. +Several libraries are domain-specific. +These domain-specific libraries serve as a catalyst for the creation of libraries tailored to other fields, sparking innovation, and expanding the reach of voice-triggered snippets. + +We divided the contents of the libraries into two categories. +One subset of libraries supports dictating about science in the Markdown cells of Jupyter notebooks, while the other subset supports writing scientific Python in code cells. +These voice-triggered snippets can also be applied to Markdown and code cells in Colab Notebooks. +While some code, such as IPython line and cell magics, is specific to Jupyter, most of the voice-triggered snippets can be used in Markdown files and Python scripts being edited in Jupyter Lab. + +Likewise, these snippets can be used in other browser-hosted text editors such as the new web version of Visual Studio Code. +This is because Voice In Plus works in most text areas of web browsers. +These means that the ample text area like that found in Write Honey website site can be used to draft documents and Python scripts with the aid of Voice In Plus snippets. +The code or text be directly edited by an advanced text editor by using the GhostText plugin to connect the web-based text area to a text editor. +Alternately, the draft document or script can be copied and saved to a file that is imported into Jupyter Lab for furtther editing. What about interaction with gradio and streamlit (like streamlit quill)? What about interacting with Python Widgets? — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC63TZUGTDGOJG4JXC3ZQU6RDAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGEZTGNZRGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:25 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +Our descriptions of these libraries include examples to illustrate how voice-triggered snippets work with automated speech recognition software. +Developers can leverage our libraries in two ways: by enhancing them with their unique commands or by using them as blueprints to create libraries from scratch. + +The libraries are made available in a modular format so the user can select the most valuable commands for their workflow. +In general, our libraries are broad in scope, so they meet the needs of most users. +Several libraries are domain-specific. +These domain-specific libraries serve as a catalyst for the creation of libraries tailored to other fields, sparking innovation, and expanding the reach of voice-triggered snippets. + +We divided the contents of the libraries into two categories. +One subset of libraries supports dictating about science in the Markdown cells of Jupyter notebooks, while the other subset supports writing scientific Python in code cells. +These voice-triggered snippets can also be applied to Markdown and code cells in Colab Notebooks. +While some code, such as IPython line and cell magics, is specific to Jupyter, most of the voice-triggered snippets can be used in Markdown files and Python scripts being edited in Jupyter Lab. + +Likewise, these snippets can be used in other browser-hosted text editors such as the new web version of Visual Studio Code. +This is because Voice In Plus works in most text areas of web browsers. +These means that the ample text area like that found in Write Honey website site can be used to draft documents and Python scripts with the aid of Voice In Plus snippets. "This means that" instead of "These means that" — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC3CB2URFFHJDY64733ZQU6URAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGEZTKNBVGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:41 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +For example, the voice command *line majic run* inserts `%run`; it is used to run script files after supplying the name of the script file as also shown in {fig}`quiz`. + +### Interactive quiz + +We developed a [quiz](https://github.com/MooersLab/voice-in-basics-quiz) to improve recall of the voice commands. +These quizzes are interactive and can be run in the terminal or in Jupyter notebooks {ref}`fig:quiz`. +The latter can store a record of your performance on a quiz. + +:::{figure} ./images/runningQuiz.png +:label: fig:quiz +:width: 130% +An example of an interactive session with a quiz in a Jupyter notebook. The code for running the quiz was inserted into the code cell with the voice command `run voice in quiz`. The quiz covers a range of voice commands, including [specific voice commands covered in the quiz]. +::: + +To build long-term recall of the commands, you must take the quiz five or more times on alternate days, according to the principles of spaced repetition learning. +These principles were developed by the German psychologist Hermann Ebbinghaus near the end of the 19th Century. You should give some citation for this discussion. It's probably in a textbook. — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC5LKCH44P6ILZDOYRDZQVAQHAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGE3DMMBZHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:41 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +These quizzes are interactive and can be run in the terminal or in Jupyter notebooks {ref}`fig:quiz`. +The latter can store a record of your performance on a quiz. + +:::{figure} ./images/runningQuiz.png +:label: fig:quiz +:width: 130% +An example of an interactive session with a quiz in a Jupyter notebook. The code for running the quiz was inserted into the code cell with the voice command `run voice in quiz`. The quiz covers a range of voice commands, including [specific voice commands covered in the quiz]. +::: + +To build long-term recall of the commands, you must take the quiz five or more times on alternate days, according to the principles of spaced repetition learning. +These principles were developed by the German psychologist Hermann Ebbinghaus near the end of the 19th Century. +They have been validated several times by other researchers. +Spaced repetition learning is one of the most firmly established results of research into human psychology. + +Most people need discipline to carry out this kind of learning because they have to schedule the time to do the follow-up sessions. +Instead,t people will find it more convenient to take these quizzes several times in 20 minutes before they spend many hours utilizing the commands. You mean "Instead, people will find it" — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC7D54ZPYJHH4RKDCYLZQVAS7AVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGE3DOMZRHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:42 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +These text areas include those of web-based email software and online sites that support distraction-free writing like [Write Honey](https://app.writehoney.com). +These text areas also include the Markdown and code cells of Jupyter notebooks and other web-based computational notebooks. +Voice In also works in plain text documents opened in Jupyter Lab for online writing. +It also works in the web-based version of [VS Code](https://vscode.dev/). +Voice In will not work in desktop applications that support the editing of Jupyter notebooks, such as the *JupyterLab* application, the *nteract* appication, and external text editors, such as *VS Code*, that support the editing of Jupyter notebooks. +*Voice In Plus* is limited to web browsers, whereas other automated speech recognition software can also operate in the terminal and at the command prompt in GUI-driven applications. + +*Voice In Plus* is very accurate with a word error rate that is well below 10\%. +Like all other dictation software, the word error rate depends on the quality of the microphone using used. +*Voice-In Plus* can pick out words from among background ambient noise such as load ventilation systems, traffic, and outdoor bird songs. + +The language model used by *Voice-In Plus* is quite robust in that dictation can be performed without an external microphone. +We found no reduction in word error rate when using a high-quality Yeti external microphone. +Our experience might be a reflection of our high-end hardware and may not transfer to low-end computers. + +Because of the way*Voice-In Plus* is set up to utilize the Speech-to-Text feature of the Google API, there is not much of a latency issue. space between way and Voice in Plus — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC5SMIC3JEW4PIXUWVTZQVAUTAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGE3DOOJUG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:44 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +You have to pay attention to the progress of the transcription if you want all of your spoken words captured. + +If the transcription halts, it is best to deactivate the plugin, activate it, and resume the dictation. +Great care must be taken to pronounce the first word of the sentence loudly so that it will be recorded; otherwise, this first word will likely not be recorded. +This problem of omitted words is most acute when there has been a pause in the dictation. + +The software does not automatically insert punctuation marks. +You have to vocalize the name of the punctuation mark where it is required. +You also have to utilize the built-in new-line command to start new lines. +We have combined the period command with the new line command to create a new command with the voice trigger of `new sentence`. + +You have to develop the habit of using this command if you like to write one sentence per line. +This latter form of writing is very useful for first drafts because it greatly eases the shuffling of sentences in a text editor during rewriting. +This form of writing is also very compatible with version control systems like git because the changes can be tracked by line number. + +The practical limit of on the number of commands it set by the trouble you are willing to tolerate in scrolling up and down the list of commands. you mean "is set by" rather than "it set by" — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC7TBXUJBJCLOT7EYMDZQVA5TAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGE3TCOBWHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

On Fri, Aug 9, 2024 at 5:45 PM Charles David Lindsey < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In papers/blaine_mooers/main.md <#934 (comment)> : > +The second extension, [jupyter-voice-comments](https://github.com/Banpan-Jupyter-Extensions/jupyter-voice-comments), relies on the DaVinci large language model to make comments in Markdown cells and request code fragments. +This program requires clicking on a microphone icon frequently, which makes the user vulnerable to repetitive stress injuries. +The third extension is [jupyter-voicepilot](https://github.com/JovanVeljanoski/jupyter-voicepilot). +Although the extension's name suggests it uses GitHub's Copilot, it uses whisper-1 and ChatGPT3. +This extension requires an API key for ChatGPT3. +The robustness of our approach is that the *Voice-In Plus* should work in all browser-based versions of Jupyter Lab and Jupyter Notebook. + +### Coping with the imperfections of the language model + +One aspect of speech-to-text that it is important to bring up is persistent errors in transcription. +These persistent errors may be due to the language model having difficulties interpreting your speech. +For example, the language model often misinterprets the word *write* as *right*'. +Likewise, the letter *R* is frequently returned as *are* or *our*'. +The remedy for these situations is to map the misinterpreted phrase to the intended phrase. + +This remedy might be the best that can be done in for those users who are from a country that is not represented by the selection of English dialects available in Voice In Plus. "This remedy might be the best can be done in for those" should be "This remedy might be the best can be done for those" — Reply to this email directly, view it on GitHub <#934 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZEC5KUT6FEGEUARQ6MBLZQVBCLAVCNFSM6AAAAABIT6HN2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEMZRGE3TIMBXGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>